Computer systems and services that serve electronic content usually index the content using indexes that are specific to the type of content being served. Therefore, an index for a web server will be designed and optimized to locate and serve web pages (e.g., .html files), an index for an ad server will be optimized to select and serve advertisements (e.g., images), a file server may be optimized to locate and serve documents, and so on. An index designed for one type of content generally cannot be used effectively with other types of content.
Also, specialized indexes generally do not permit simultaneous reading and writing. Therefore, whenever the index must be maintained (e.g., to add or remove an entry), the entire index or a portion of the index may be temporarily locked or taken offline, and can't be used to locate and serve content during the maintenance. Depending on how long it is locked for writing, searches may be delayed for unacceptable periods of time or may not be executable against the full collection of data.
Further, maintaining a specialized index may be time-consuming if multiple operations must be applied during a maintenance cycle. For example, in between maintenance cycles multiple updates to the indexed data may be queued until the next cycle commences, at which time they are all applied in serial fashion. The duration of the maintenance (e.g., and the data's unavailability for searching) may be proportional to the number of updates that were received.
Yet further, scanning or searching a specialized index can be relatively slow when entries in the index contain some or all of the semantic content (e.g., in textual format) that must be read, parsed and compared with some target parameters in order to determine which index entries are relevant to the current search. In particular, an index entry representing a web page, an advertisement, a news story or other type of item may contain textual content of the item. When a query is executed against the index, the textual content must be compared with the query, which can involve execution of a relatively slow pattern-matching algorithm for comparing text.
Even further, a specialized index for serving just one type of content may not be optimized to find the best or most valuable content first. Instead, such indexes may be organized such that the entire index needs to be searched in order to ensure that the most valuable content is located.